The null-hypothesis significance-test procedure (NHSTP) is defended in the context of the theory-corroboration experiment, as well as the following contrasts: (a) substantive hypotheses versus statistical hypotheses, (b) theory corroboration versus statistical hypothesis testing, (c) theoretical inference versus statistical decision, (d) experiments versus nonexperimental studies, and (e) theory corroboration versus treatment assessment. The null hypothesis can be true because it is the hypothesis that errors are randomly distributed in data. Moreover, the null hypothesis is never used as a categorical proposition. Statistical (...) significance means only that chance influences can be excluded as an explanation of data; it does not identify the nonchance factor responsible. The experimental conclusion is drawn with the inductive principle underlying the experimental design. A chain of deductive arguments gives rise to the theoretical conclusion via the experimental conclusion. The anomalous relationship between statistical significance and the effect size often used to criticize NHSTP is more apparent than real. The absolute size of the effect is not an index of evidential support for the substantive hypothesis. Nor is the effect size, by itself, informative as to the practical importance of the research result. Being a conditional probability, statistical power cannot be the apriori probability of statistical significance. The validity of statistical power is debatable because statistical significance is determined with a single sampling distribution of the test statistic based on H0, whereas it takes two distributions to represent statistical power or effect size. Sample size should not be determined in the mechanical manner envisaged in power analysis. It is inappropriate to criticize NHSTP for nonstatistical reasons. At the same time, neither effect size, nor confidence interval estimate, nor posterior probability can be used to exclude chance as an explanation of data. Neither can any of them fulfill the nonstatistical functions expected of them by critics. (shrink)

Some important meta-theoretical insights about experimental psychology are integrated into the "conjectures and refutations" framework in order to reinforce a realist's view of scientific methodology. Some issues which may be difficult for the realist's position are discussed. It is argued that there is no need for the evidential observation to mimic the phenomenon of interest; such a mimicry may even be counter-productive. A case is also made that questions about ecological validity are not relevant to the rationale of experimentation.

The rhetoric-analytic critique of experimental psychology owes its apparent attractiveness to (a) some erroneous ideas about cognitive psychology and the rationale of experimentation, (b) the failure to distinguish between prior data and evidential data vis-à-vis the to-be-corroborated explanatory theory, and (c) evidential data owes their identity to a theory that is independent of the theory being tested. Theories in cognitive psychology are accepted because they can withstand concerted efforts to falsify them.

Ambiguous data obtained by deception say nothing about social behavior. A balanced social psychology requires separating statistical hypotheses from substantive hypotheses. Neither statistical norms nor moral rules are psychological theories. Explanatory substantive theories stipulate the structures and processes underlying behavior. The Bayesian approach is incompatible with the requirement that all to-be-tested theories be given the benefit of the doubt.

Entertaining diverse assumptions about empirical research, commentators give a wide range of verdicts on the NHSTP defence in Statistical significance. The null-hypothesis significance- test procedure is defended in a framework in which deductive and inductive rules are deployed in theory corroboration in the spirit of Popper's Conjectures and refutations. The defensible hypothetico-deductive structure of the framework is used to make explicit the distinctions between substantive and statistical hypotheses, statistical alternative and conceptual alternative hypotheses, and making statistical decisions and drawing theoretical (...) conclusions. These distinctions make it easier to show that H0 can be true, the effect size is irrelevant to theory corroboration, and “strong” hypotheses make no difference to NHSTP. Reservations about statistical power, meta-analysis, and the Bayesian approach are still warranted. (shrink)

That Haig and Sohn find the hypothetico-deductive approach wanting in different ways shows that multiple conditional syllogisms are being used in different stages of theory corroboration in the Popperian approach. The issues raised in the two commentaries assume a different complexion when certain distinctions are made.